Introduction

• In recent years, expansion of the expert survey methodology, a method for measuring complex political concepts (Marks et al. 2007).
• On party ideology, recent datasets include over 1,000 observations and > 150 countries, e.g. V-Party (Lührmann et al. 2020) and GPS (Norris 2020).
• But, to what extent broader coverage in expert surveys comes at the cost of reduced precision?

Objectives

• Investigate whether experts exhibit greater disagreement when evaluating parties:
  • 1. in less democratic institutions;
  • 2. relatively small
• Contribute to the literature on expert survey methodology.

Expert surveys (I)

Since social phenomena are complicated to measure, some strategies rely on surveying experts: individuals with specialized knowledge of one or more countries who can synthesize multiple sources of information (Hooghe et al. 2010).

• Expansion of cross-national datasets, from limited geographic coverage (Castles and Mair 1984) to nearly global reach (Norris 2020).
• Currently, similar multi-rater design (Maestas 2018) and aggregation methods (Graefe et al. 2014).
• Variation across question wordings, response scales, and pools of experts.

Expert surveys (II)

Expert surveys face two primary types of error: systematic and random error (Maestas 2018):

• Systematic (bias / accuracy): Non-independent assessments. Consistent in the experts’ responses, leading to a deviation from the true estimate. Very problematic because it skews the results.
• Random (noise / precision): Independent, but different assessments. Variability in the experts’ responses that does not follow a specific pattern. Unproblematic if truly independent, but in most cases, the degree of independence is unknown.

Expert surveys (III)

Sources of random error (Budge 2000; Martínez-Coma and Van Ham 2015):

• Individual cognitive and judgment biases (e.g. experts might rely in different heuristics, have ideological leanings or vary in their knowledge).

• Political and social context under which the evaluation is made (e.g. autocracies can punish certain political orientations).

• Some concepts, due its nature, are inherently complex (e.g. political ideology, government effectiveness, political polarization).

Theoretical arguments (I)

Regarding context’s characteristics, random biases have been extensively explored in Europe (Bakker et al. 2014), but less outside of it. We can expect that the type of political regime affects:

• Information asymmetries faced by experts: more information in democracies, less in autocracies.
• Ideological leaning of the assessments: more leaning in autocracies, less in democracies.

Theoretical arguments (II)

About the unit’s characteristics, non-significant relationship between party size and experts’ disagreement (Marks et al. 2007; Steenbergen and Marks 2007). But:

• More information (resources, media attention) on large parties than small parties.

• But more ambiguity in catch-all parties - strategies more difficult to evaluate with precision.

• Thus, more precision in “medium” parties?

Theoretical arguments (III)

But though small party units may lack of resources, some can be easier to evaluate if have enough stability and longevity in the party system. One can difference between:

• Statewide small parties, often with shorter lifecycles, being more difficult to evaluate with precision.
• Regional (small) parties, typically tied to specific regionally concentrated groups, more stable, being easier to evaluate.

Theoretical arguments (III)

About the policy dimension, not yet.

Methodology (I)

• Data from 168 countries and 10041 party-policy observations derived from the V-Party (Lührmann et al. 2020), CHES (Bakker et al. 2020) and GPS (Norris 2020).
• Similar time period, aggregation, and dimensions (next).
• Excluded assessments conducted with only one expert (from 2 to 84).
• Controling for the fixed effects of the dataset.

Methodology (II)

• Dependent variable: Disagreement among experts, using the standard deviation, a common measure of uncertainty (Lührmann et al. 2020; Bakker et al. 2020; Coppedge et al. 2021).

Methodology (III)

• Party size (IV1): Operationalized with the percentage of votes received by the party in the national parliament (Marks et al. 2007; Steenbergen and Marks 2007).

Methodology (IV)

• Regional party (IV2): Defined as political parties that seek representation at the substate level (93 obs.). Data from Massetti and Schakel (2021) and Nordsieck (2019).

Figure 4: State and non-state wide parties

Methodology (V)

• Democracy (IV3): Liberal democracy (continuous) from V-Dem (Coppedge et al. 2021) and dichotomous indicator from DCD (Boix, Miller, and Rosato 2013).

Figure 5: Continuous (V-Dem) and dichotomous (DCD) measures of democracy

Methodology (VII)

• Controls: Expertise is calculated with the ratio of NAs per party-question (Martínez-Coma and Van Ham 2015).

Figure 6: Pairwise correlations grid

Results (I): Bivariate analysis

• Regime type and SD: Slightly negative, seemingly inverted U-curved relationship (specially for small parties).

Results (II): Bivariate analysis

• Size and SD: Slight U-curved association, irrespective of the regime. Less disagreement for medium-sized parties.

Results (III): Bivariate analysis

• Regional and SD: High disagreement on statewide parties, much less on regional parties, specially in autocracies.

Conclusion

• Experts are more precise in democracies, though this can be potentially attributed to higher levels of economic development.
• Democracies and autocracies seem to follow very different patterns, which need to be explored in depth.
• If anything, party size follows a non-linear relationship with noise.
• Experts show minimal disagreement when locating regional (small) parties, ceteris paribus, especially in comparison to small statewide parties.

Work to do

• Explore non-linear relationships (party size and regime type).
• Consider new controls: member of government, internal dissent, or policy-dimension FE.
• Look at variations across policy dimensions.